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Accuracy  of  Technical  Estimates 
In  Industrial  Research  Planning 

by 

Dennis  L.  Meadows*  and  Donald  G,  Marquis* 

Abstract 
To  understand  the  estimate  errors  which  must  be  accommodated  by  formal 
project  selection  models,  this  study  investigates  the  relationships  among 
initial  estimates  and  actual  outcomes  in  industrial  product-development 
programs.   The  following  preliminary  conclusions  are  based  on  data  from 
five  commercial  laboratories: 

1.  The  estimate  errors  employed  in  current  selection  procedures  can 
lead  management  to  expend  more  than  50  percent  of  the  firm's  development 
resources  on  technically  or  commercially  unsuccessful  projects, 

2.  Unsuccessful  projects  tend  to  incur  greater  cost  overruns  and  to 
cost  more  on  the  average  than  do  projects  resulting  in  commercially  success- 
ful products. 

3.  Current  estimates  are  not  sufficiently  accurate  to  justify  the  use 
of  formal  selection  formulas.  The  most  accurate  estimates  of  project  cost 
explain  only  25%  of  the  variance  among  actual  costs.  Initial  estimates  of 
probability  of  technical  or  commercial  success  do  not  usefully  distinguish 
among  projects  which  have  different  degrees  of  success. 

U.   The  average  magnitude  of  cost  overruns  and  the  rate  of  commercial 
success  depend  in  part  on  the  source  of  the  project  idea. 
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+  Professor  of  Manetgement,  Sloan  School  of  Management,  M.I.T. 


Introduction 

One  fact  must  disturb  anyone  interested  in  improving  the  administration  of 
industrial  research  and  development  programs.  Although  no  aspect  of  laboratory 
administration  has  received  more  attention  in  the  management  science  literature 
than  has  project  selection  procedure,  there  appears  to  be  almost  no  implemen- 
tation of  the  formal  selection  models  which  have  been  proposed.  Baker  and 

Pound's  review  of  the  project  selection  literature  included  over  80  different 

1 
formal  models,  all  proposed  for  use  in  evaluating  development  project  proposals. 

Their  study  of  implementation  found  that  only  six  laboratories  out  of  thirty-five 
have  experimented  with  even  one  of  the  models.  None  of  the  thirty-five  administra- 
tors were  currently  using  such  a  model  in  their  laboratories,   A  survey  of  R&D 
policy  in  U.S.  chemical  companies  in  Chemical  and  Engineering  News  also  found 

a  low  reliance  on  formal  models  in  the  selection  of  new  product  development  pro- 
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jects.   The  prevalent  attitude  was  characterized  by  one  of  the  administrators 

interviewed  in  the  study. 

We've  played  around  with  several  formulas, 
but  we  decided  that  they  give  no  more  in- 
sight than  does  mature  Judgment  by  manage- 
ment , 

As  a  nation  we  invest  about  $25  billion  annually  in  R&D  activity.  Many  firms 

spend  from  5^  to  10^  of  their  gross  sales  income  on  R&D  programs.  Most  of  this 

money  is  invested  in  product  development  and  improvement  programs ,  the  phase  of 

R&D  for  which  formal  selection  models  have  been  specifically  proposed.  Management 

science  models  have  made  an  important  contribution  to  other  areas  of  management. 

If  we  aspire  to  increase  the  effectiveness  of  our  investment  in  new  technology, 

it  is  important  to  understand  why  current  efforts  by  management  scientists  in 

developing  project  selection  models  have  borne  so  little  fruit.  The  reasons 

underlying  that  failure  must  have  implications  for  the  design  and  use  of  such  models, 
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Estimate  Error 

In  their  stuciy  of  implementation,  Baker  and  Pound  conclude  that  formal  models 
have  not  been  implemented  because  they  have  not  been  specifically  designed  to  fit 
the  intrinsic  structure  of  the  project  selection  decision.   One  important  aspect 
of  that  problem  is  the  models'  xiniversal  failure  to  recognize  and  take  account  of 
any  errors  in  the  estimates  which  constitute  their  inputs » 

The  uncertainty  in  estimates  is  the  difficulty  which  most  laboratory  directors 

first  note  in  any  formal  models   Again  the  C&EH  study  provides  a  characteristic 

remark : 

Formulas  mean  nothings  They  are  just  an  accumulation 
of  approximations  and  assumptions,  which  may  not  have 
any  relation  to  reality. 

Of  course  there  are  other  important  obstacles  to  the  use  of  the  current  formal 

models . 

-  Each  development  project  must  typically  serve  many  incommensurate  and  often 
conflicting  goals.   It  is  neither  valid  not  useful  to  evaluate  the  project's 
total  contribution  along  only  one  dimension ^  yet  the  output  of  most  models 
is  a  single  index  of  value, 

-  The  laboratory  is  a  complex  social  system  with  many  subtle  influences  on 
individual  performance.   The  procedure  which  assigns  resources  among 
competing  projects  is  a  major  determinant  of  employee  satis  faction  r.   Thus 
personnel  considerations  must  enter  into  the  selection  and  use  of  any  pro- 
ject evaluation  model.   The  management  science  models  currently  available 
do  not  consider  such  factors. 

-  The  costs  and  returns  of  any  given  development  project  depend  in  complex 
ways  upon  the  nature  of  other  projects  undertaken  by  the  laboratory.   The 
great  majority  of  all  formal  models  evaluate  each  project  individually, 
ignoring  the  interdependencies. 

Of  trie  obstacles  listed  above,  estimate  error  is  the  most  important  factor  in  the 

failure  of  formal  project  selection  models.   It  is  primarily  through  estimate 

errors  that  the  other  three  structural  problems  affect  the  selection  decision; 

-  Even  models  designed  to  evaluate  a  project  proposal  along  c^iveral  different 
dimensions  must  rely  upon  estimates  of  the  relevant  attributes,  and 
errors  in  those  estimates  will  limit  the  utility  of  the  models, 

-  When  the  selection  rule  does  not  fit  its  organizational  milieu, 
experiments  suggest  that  employees  can  respond  by  deliberately 
biasing  the  estimates  used  in  the  decision  rule, 


-3- 

-  Interdependencies  among  the  costs  and  payoffs  of  different 
projects,  which  are  ignored  by  the  selection  rule,  vill 
manifest  themselves  through  outcomes  which  differ  from  those 
anticipated,  ie.  through  estimate  errors. 

In  short,  estimates  are  the  vital  link  between  any  selection  procedure  and 
the  organization  which  employs  it.   All  organizational  and  technical  factors  which 
must  enter  into  the  design  of  a  formal  project  evaluation  procedure  and  which 
determine  its  effectivenss  can  be  expressed  in  terms  of  their  impact  on  the  differ- 
ence between  estimated  and  actual  project  outcomes.   Laboratory  managers  are  of 
course  only  interested  in  the  actual  outcomes  of  their  projects.  However,  the 
rules  they  employ  in  allocating  resources  among  alternative  projects  must  be  based 
primarily  on  estimates  of  those  outcomes.  Social  and  technical  factors  can  both 
lead,  in  theory,  to  deviations  between  the  initial  estimates  and  the  actual  outcomes. 

The  experience  of  laboratory  administrators  appeurently  suggests  that  the  errors 
introduced  by  these  factors  are  large  enough  to  prevent  current  formal  models  from 
being  useful  in  the  selection  decision.   It  is  thus  naive  to  propose  any  model  and 
anticipate  its  implementation  in  the  laboratory  without  first  understanding  the 
relationships  between  estimated  and  actual  project  outcomes. 

Errors  and  Current  Models 
Current  selection  formulas  almost  completely  ignore  the  difficulties  intro- 
duced by  estimate  errors.  Where  the  problem  of  uncertainty  is  even  aclcn  owl  edged, 
its  solution  is  implicitly  left  to  the  administrator. 

The  procedures  developed  nere  can  be  applied  where  projects  can 
be  classified  into  groups  which  have  different  probabilities  of 
success  and  where  the  necessary  estimates  can  be  obtained  with 
an  acceptable  degree  of  accuracy. 

No  operational  criteria  are  given  for  determining  exactly  when  or  how  such  esti- 
mates can  be  generated.  Nor  is  there  a  discussion,  typically,  of  what  constitutes 
"an  acceptable  degree  of  accuracy".   One  author  even  concluded  after  a  survey  of 
formal  project  selection  models  and  a  discussion  of  the  estimates  upon  which  they 
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were  based,  that  "measuring  the  accuracy  of  these  estimates  would  be  difficult 
if  not  impossible."' 

Measurement  of  estimate  errors  is  indeed  difficulty  as  Marshall  and  Meckling 
discovered  while  assessing  the  uncertainty  inherent  in  the  development  of  Air 
Force  weapon  systems.   Problems  are  introduced  by  changes  in  the  price  level, 
in  the  magnitude  of  the  production  run,  and  in  the  objectives  of  the  project. 
The  widespread  use  of  ambiguous  estimates  which  do  not  explicitly  state  assumptions 
about  interdependent  variables  also  complicates  measurement  of  errors »   However, 

Q 

Marshall's   study   of  cost  and  time   overruns,   and  that   of  Peck  and  Scherer,^   did 

prove  it  possible  to  derive  useful  conclusions  from  such  research. 

The  most  important  conclusion  of  both  studies  corroborates  the  subjective 

Judgment  of  laboratory  administrators.  Estimate  errors  can  be,  and  often  are, 

very  large.   Marshall  and  Meckling  report  the  average  ratio  of  final  cost  to  the 

earliest  estimate  (adjusted  for  changes  in  output  and  in  price  level)  as  follows: 

9  fighter  planes  1.7 

3  bombers  2.7 

k   cargoes  and  tankers  1.2 

6  missiles  U.l 

Peck  and  Scherer  reported  an  average  ratio  of  3.2  for  12  planes  and  missile  de- 
velopments.  Schedule  slippage  on  10  programs  was  consein;-atively  determined  by 
Marshall  and  Meckling  to  range  from  2  years  to  5  years,  with  an  average  ratio 
of  final  completion  time  to  earliest  estimate  of  1.5.   The  corresponding  ratio 
for  the  12  programs  examined  by  Peck  and  Scherer  was  1.36. 

If  industrial  development  programs  are  also  characterized  by  the  uncertainties 
found  in  these  military  studies,  none  of  the  formal  models  currently  proposed  can 
provide  much  assistance  to  management  in  selecting  among  projects.  However, 
government  programs  characteristically  differ  from  industrial  projects  in  magni- 
tude, in  organizational  environment,  and  in  the  amount  of  technological  advance 
required.  Thus  it  is  difficult  to  assess  the  relevance  of  the  findings  to 
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coinmercial  programs. 

Mansfield  has,  undertaken  the  only  detailed  study  of  estimated  and  actual 

10 
outcomes  in  a  large  sample  of  industrial  product  development  projects.    How- 
ever, he  obtained  no  data  on  the  commercial  outcomes  of  the  projects,  and  his 
measurement  of  probability  estimating  performance  is  not  based  on  the  meaning 
which  laboratory  administrators  typically  assign  to  those  estimates. 

We  have  discovered  no  other  published  study  assessing  the  nature  of  errors 
in  the  estimates  employed  to  select  among  alternative  commercial  development 
projects.   After  their  extensive  review  of  the  literature  Baker  and  Pound  reported, 
"no  references  were  found  which  attempted  to  give  data  and  discuss  the  actual  role 
of  uncertainty  in  R&D  project  selection."^ 

It  obviously  is  impossible  to  eliminate  uncertainity  from  development  pro- 
grams, but  as  we  come  to  understand  the  influence  of  various  social  and  technical 
factors  on  estimate  error,  we  can  design  selection  procedures  which  will  best 
take  account  of  xincertainty  in  specific  laboratories.  The  present  study  is  de- 
signed to  explore  systematically  the  relation  between  estimated  and  actual  out- 
comes in  commercial  development- projects.   Large  scale  studies  of  many  more  projects 
in  both  industrial  and  defense  laboratories  are  currently  in  process. 

Data  Sources 
Data  from  five  firms  will  be  utilized  here  to  discuss  several  of  the  prelimin- 
ary findings.*  Three  of  the  laboratories  are  in  chemical  companies  engaged  pri- 
marily in  the  development,  production,  and  sale  of  industrial  intermediates.  Two 
of  the  chemical  firms,  A  and  B,  have  between  $100  million  and  $300  million  annual 
sales.  The  third  is  one  of  the  largest  firms  in  the  nation.  The  fourth  laboratory, 
called  here  the  Electronics  Laboratory,  is  maintained  by  an  instrumentation  company 


*  "a  detailed  data  appendix,  from  which  company  and  project  titles  have  been 
eliminated,  is  available  on  request  from  the  Program  Office,  Md.T.  E52-539, 
Cambridge,  Mass,  02139." 

We  acknowledge  the  assistance  of  Kenneth  Seltzer  in  compiling  and  analyzing 
a  portion  of  the  data. 
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to  engage  in  work  of  interest  to  NASA  and  the  Department  of  Defense.  The  fifth 
set  of  project  data  is  taken  from  Mansfield's  article  describing  his  study  of  a 
central  research  unit  in  a  prominent  equipment  manufacturero 

Data  from  each  of  the  five  laboratories  were  obtained  from  three  sources. 
The  initial  estimates  of  project  cost,  probability  of  technical  success,  and 
probability  of  commercial  success  were  obtained  from  project  evaluation  forms 
completed  at  the  time  the  project  was  first  approved  for  funding.  Accounting 
records  supplied  the  actual  costs,  and  the  leader  of  each  project  indicated 
whether  the  project  had  eventually  been  technically  and/or  commercially  success- 
ful. 

Completed  projects  are  divided  into  four  categories  according  to  their  out- 
comes . 

Miscellaneous  failure:  The  project  was  closed  out  because  of  manpower 
shortages,  changes  in  the  market  objectives  of  the  firm,  or  other 
non-technical  reasons.   The  project  did  not  result  in  a  product. 

Technical  failure:   The  project  was  closed  out  when  unforeseen  technical 
difficulties  prevented  the  development  program  from  producing  the 
desired  product. 

Commercial  failure:   The  project  did  produce  the  product  initially  desired, 
but  it  did  not  produce  any  sales  income  for  the  firm. 

Commercial  Success:   The  project  was  technically  successful,  and  the  pro- 
duct did  produce  sales  income  for  the  firm. 

The  classifications  of  technical  and  commercial  success  are  dichotomous. 

With  the  exception  of  Chemical  Laooratory  B,  no  attempt  was  made  to  rank  products 

on  the  basis  of  their  technical  or  commercial  performance.  The  above  definitions 

do  not  apply  to  the  projects  in  Chemical  Laboratory  C  where  the  estimated  and 

actual  costs  for  only  one  yeeur's  development  on  each  project  were  available. 

Conclusions 

l)   Current  laboratory  selection  procedures  typically  expend  more  than  30% 

of  the  firms  development  resources  on  projects  which  do  not  produce 

commercially  successful  products.  (Table  1) 


-7- 


Table  1 

LABORATORY  INVESTMENT  IN  UNSUCCESSFUL  PROJECTS 

%   TOTAL  COST 


Chem 

Chem 

Project  Outcome 

Lab  A 

Lab  B 

Electronics 

Lab 

Equip.    Lab 

Miscellaneous 
Failure 

Technical 
Failure 

li5^(^) 

18^ 

lOiS 

25% 

27% 
20% 

Commercial 
Failure 

31% 

51!^ 

18% 

51%C2) 

Commercial 
Success 

18^ 

31^ 

kl% 

(1)  Project  leaders  responded  to  the  question,  "Did  the  project  achieve  its 
technical  objectives?"  Thus  no  distinction  was  made  between  technical 
and  miscellaneous  failures. 

(2)  Mansfield  did  not  obtain  information  on  the  commercial  outcome  of  the 
projects  in  his  study.  Thus  it  is  impossible  to  distinguish  between 
those  projects  which  were  commercially  successful  and  those  which  were 
only  technically  successful.   Rounding  errors  prevent  percent  total 
actual  cost  from  equaling  100%. 

Tne  total  cost  of  unsuccessful  projects  is  a  qualitative  meaisure  of  uncertainty. 

The  remaining  conclusions  relate  technical  and  organizational  factors  to  estimate 

uncertainty. 

2)  There  is  a  pronounced  tendency  for  technically  and  commercially  xuisuccess- 
ful  projects  as  a  group  to  incur  greater  cost  overruns  than  commercially 
successful  projects.   (Table  2) 

When  projects  are  grouped  according  to  their  outcomes,  the  ratio  of  total 
actual  to  total  estimated  costs  for  the  unsuccessful  projects  of  each  laboratory 
is  found  to  be  greater  than  for  the  corresponding  group  of  successful  projects, 

3)  Technically  and  commercially  vinsuccessful  projects  cost  more  on  the 
average  than  commercially  successful  projects.  (Table  2) 

k)  There  is  a  low  correlation  between  the  actual  and  estimated  values  of 
cost,  probability  of  technical  success,  and  probability  of  commercial 
success  figures  employed  in  selecting  among  projects.  (Figures  1-U) 
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Table  2 
AVERAGE  PROJECT  COST  AND  COST  OVERRUN  BY  PROJECT  OUTCOME 


Laboratory  Project  Outcome 

# 

Total  Actual  Cost 

Average  Cost 

Projects 

Total  Estimated  Cost 

Per  Project 

Chem  Lab  A 

Miscellaneous  &  Technical 

Failure 

7 

2.55 

$20,780 

Commercial  Failure 

11 

1.28 

$lk ,200 

Commercial  Success 

12 

0.9U 

$  6,U30 

Chem  Lab  B 

Miscellaneous  &  Technical 

Failure 

k 

3.81* 

$  2,690 

Commercial  Failure 

12 

U.25 

$  2,520 

Commercial  Success 

17 

1.27 

$  1,070 

Electronics  Laboratory- 

Miscellaneous  Failure 

7 

1. 

k  man-yrs 

Technical  Failure 

5 

k. 

8  man-yrs 

Commercial  Failure 

8 

2. 

2  man-yrs 

Commercial  Success 

22 

2, 

1  man-yrs 

Equipment  Laboratory 

Miscellaneous  Failure 

12 

.96 

(1)   1.5^ 

Technical  Failure 

7 

1.25 

2.9!? 

Commercial  Failure  & 

Commercial  Success 

20 

0.96 

2.5? 

(1)   Costs  are  given  as  a  percentage  of  the  total  laboratory  expenditure 
on  the  U5  projects. 


Cost  Estimates 

In  Chemistry  Laboratories  A  and  B,  the  cost  of  successfully  meeting  the 
technical  requirements  is  predicted  for  each  project  before  deciding  whether  it 
should  be  funded.  The  actual  costs  of  projects  which  were  technically  success- 
ful can  thxis  be  compared  with  the  initial  estimates  to  measure  the  error  in  those 
estimates.   Figures  #1  and  #2  compare  estimated  with  actual  costs  for  each  of  the 
two  organizations. 

The  actual  costs  are  generally  found  to  differ  substantially  from  those 
initially  predicted.  Where  there  are  deviations  of  this  type,  it  has  been  suggested 


-9- 


o 
o 
o 


EH 

o 
o 


EH 

O 

< 


100 
70h 

50 

^0 

30 

20 


10 
8 

6 
5 

3 

2 


V 


•  #^     /  S 


2    3   ^  5  6   8  10      20   30  i^O  50   70  100 
ESTIMATED  COST  ($000) 


ESTIMATED  vs.  ACTUAL  COSTS  FOR  23  TECHNICALLY 
SUCCESSFUL  PROJECTS  -  CHEMICAL  LABORATORY  A 


Figure   1 
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Figure  2 


that  a  linear  formula  in  the  form;  -11- 

Actual  Cost  =  A  +(B) (Estimated  Cost) 

be  employed  to  modify  the  initial  estimates,  correcting  for  the  errors  inherent 

12 
in  them.    It  is  easy  to  determine  statistically  the  "best"  linear  formula 

for  any  given  set  of  historical  data,  but  many  factors  limit  the  predictive 

ability  of  such  an  equation  by  causing  some  deviation  of  the  actual  costs  from  the 

modified  estimateso  The  correlation  coefficient,  r,  measures  this  deviation^  The 

2 
quantity  r  indicates  what  percent  of  the  variance  in  actual  costs  is  explained 

by  the  modified  estimates.   If  r  (consequently  r  )  is  low,  the  estimates  will  be 
of  little  use  in  predicting  actual  costs.   The  correlation  coefficient  for  each 
set  of  projects  is  given  on  the  corresponding  figureo  The  most  accurate  estimates, 
those  in  Chemical  Laboratory  A  explain  only  25%   of  the  variance  in  the  actual  pro- 
ject costs  even  when  they  are  modified  with  the  best  linear  correction  formula  for 
that  particular  set  of  data: 

Actual  Cost  =  $596  +  I.U5  (Estimated  Cost) 
In  Chemistry  Laboratory  C  project  costs  are  predicted  for  only  one  year  at  a  time„ 
The  uncertainty  in  annual  cost  estimates  should  be  substantially  less  than  that 
in  the  estimates  of  total  project  costs.  The  estimates  are  more  accurate  than 
those  found  in  the  other  chemistry  laboratories.  However,  the  estimates  still 
account  for  no  more  than  3l7»   to  lh%   of  the  variance  in  actual  costs ^ 

Probability  Estimates 

The  staffs  of  Chemical  Laboratory  A  and  the  Equipment  Laboratory  estimate 
the  probability  of  technical  success  and  the  probability  of  commercial  success 
of  each  development  project  before  deciding  whether  it  should  be  funded,.  The 
estimated  values  are  employed  by  management  as  if  they  characterized  a  binomial 
process.   For  example,  of  all  the  projects  which  receive  the  initial  estimate 
PTS  =  O08O,  80>S  are  expected  to  be  technically  successful. 

Probability  estimate- errors  are  more  difficult  to  measure  than  cost  estimate 
errors,   A  numerical  overrun  ratio  can  be  employed  to  indicate  the  error  in  an 
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estimated  cost  figure.   There  is  no  equivalent  measure  of  probability  estimate 
error.  When  a  coin  lands  with  cme  side  up  on  eight  of  ten  losses,  it  does  not 
necessarily  disprove  the  initial  estimate  that  the  probability  of  that  side 
landing  up  is  0,50.,  Similarly,  a  deviation  between  the  fraction  of  projects 
which  actually  succeed  and  the  probability  of  technical  success  initially  assigned 
a  set  of  projects  does  not  necessarily  prove  the  initial  estimate  to  have  been  in 
error.  A  deviation  of  the  fraction  actually  successful  from  that  expected  on 
the  basis  of  the  initial  estimate  is  particularly  likely  to  occur  when  there  are 
only  a  few  projects  in  the  group.   Thus  only  statistical  measures  of  probability 
estimate  error  are  possible. 

To  measure  the  accuracy  of  the  probability  of  technical  success  estimates  in 
Chemistry  Laboratory  A,  all  projects  were  divided  into  groups  according  to  the 
probability  of  technical  success  value  each  project  was  initially  assignedo  The 
estimated  probability  of  technical  success  values  Eire  indicated  on  the  horizontal 
axis  of  Figure  #3,   The  fraction  of  each  group  which  actually  succeeded  was  deter- 
mined and  the  fractions  are   indicated  by  vertical  white  barso  Finally,  that  fractior 
which  was  technically  successful  was  used  to  calculate  limiting  values  of  the 
true  probability  of  success.  From  the  fraction  actually  successful  in  each  set 
of  projects,  an  interval  was  found  which  has  a  50^  probability  of  including  the 
true,  but  vmknown,  probability  of  technical  success.  These  50%  confidence  inter- 
vals are  shown  in  Figure  #3  as  shaded  bands.   Although  the  procedure  is  compli- 
cated, it  is  not  difficult  to  interpret  the  results.  If  half  the  shaded  bands 
cross  the  line  indicating  the  expected  fraction  successful,  the  initial  estimates 
have  not  been  proved  inaccurate. 

The  measurement  of  probability  of  commercial  success  differs  from  the  above 
in  only  one  respect.   Each  estimate  of  probability  of  commercial  success  is  based 
on  the  assumption  that  the  project  will  be  technically  successful.   Thus  only  the 
20  technically  successful  projects  are  included  in  the  analysis  shown  in  Figure  #i+. 
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The  results  tend  to  confirm  the  laboratory  management's  belief  that  market 

13 
estimates   are  less   reliable  than  technical  estimates.  Figures   #3  and  0k 

indicate  that  the  initial  probability  estimates  in  Chemistry  Laboratory  A  are 

not  accurate  enough  to  be  employed  in  project  evaluation  formulas. 

Mansfield  assigned  a  different  meaning  to  the  Equipment  Laboratory  probability 
of  technical  success  estimates.     Thus  his   analysis   of  estimate  error  differs   from 
that  described  above.      Nevertheless,  he  too  concluded  that,   "although  there  is   a 
direct  relationship  between  the  estimated  probability  of  success   and  the  outcome 
of  a  project,  it  is  too  weak  to  permit  very  accurate  predictions.' 

5)      The   average  magnitude   of  cost   overruns   and  the   rate   of  commercial 

success  may  differ  depending  upon  the  sources  of  the  project  idea. 
(Tables  3  ^  h) 

Cost  Overruns 

In  Chemistry  Laboratory  B  the  source  of  the  project  idea  was  recorded.      It  is 

thus  possible  to  determine  the   average   cost  overrun   of  projects   initiated  by  the 

laboratory,  by  the  marketing  staff,  or  by  a  customer.      The  results   are  summarized 

in  Table   #3. 

Table   3 

AVERAGE  COST  OVERRUN  BY  PROJECT  SOURCE 

CHEMICAL  LABORATORY  B 


Project  Idea  Source  a-TPT-ggP   Actual  Cost 

Estimated  Cost 


Laboratory  2,20 

Marketing  Department  2.02 

Customer  1.27 


The  different  overruns  in  Table  #3  probably  correspond  to  the  magnitude  of 
technological  advance  attempted  in  each  group  of  projects.   Customers  tend  to 
request  only  product  modifications,  while  laboratory  personnel  are  inclined  to 
suggest  more  difficult  technical  problems.  The  marketing  staff  would  tend  to 
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fall  between  the  two  extremes.   This  interpretation  is  supported  by  a  study  of  mili- 
tary development  programs.   Marshall  and  Heckling  classified  26  programs  into  three 
degrees  of  required  technological  advance  and  found  the  overrun  factor  to  be  1.1+ 
for  smallj  l.T  for  medium,  and  3,h   for  large  technological  advances.  ^ 

Rate  of  Success 

Projects  in  Chemistry  Laboratory  B  which  were  initiated  at  the  suggestion  of 
customers  have   a  much  greater  probability  of  commercial  success   than  projects 
originating  in  either  the  marketing  department   or  the   laboratory.      Table   #1+  gives 
the  percent  of  each  project  group  which  resulted  in  no  sales,   small,  medium,  or 
large  sales.      The  expenditure  invested  in  projects   from  each  source  is   also  included 
in  the  table.      Taken  together,  Tables   #3  and  ^U  suggest  that  the  magnitude  of 
imcertainty  associated  with  a  development  project  may  be   related  to  the  project's 
source. 


Table   k 
COMMERCIAL  OUTCOME  BY  PROJECT  SOURCE 
CHEMICAL  LABORATORY  B 


Percent 
Project  Source 

of  Projects 
Laboratory 

Marketing 

Incremental  Sales 

None 

66f, 

58^ 

Small 

11% 

lUj? 

Medium 

n% 

lk% 

Large 

0% 

looT 

lk% 

looi 

Percent  of  Total 

ko% 

ko% 

Budget  Expended 

Customer 

33^ 
33^ 
13^ 

Tool" 

20^ 
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Discussion 

The  magnitude  of  the  errors  presented  above  is  characteristic  of  all  the 
laboratories  studied,  but  the  assessment  of  its  ultimate  generality  must  await 
more  data  from  other  laboratories.  However,  each  of  the  companies  above  is 
considered  a  technical  leader  by  the  others  in  its  fields   Each  company  is  prof- 
itable and  growing.   One  cannot  dismiss  the  seemingly  poor  estimating  performance 
as  the  result  of  inept  technical  staff  or  poor  management.   Many  types  of  important 
estimates  have  not  yet  been  studied,  but  probability  of  technical  success  and  pro- 
ject cost  are  the  two  pareuneters  which  laboratory  managers  feel  they  can  estimate 
most  accurately.    Thus  we  do  not  expect  that  other  types  of  estimates  will  be 
found  more  reliable. 

Unless  the  laboratories  we  have  studied  are  atypical,  estimate  error  must  be 
the  first  concern  of  laboratory  managers  interested  in  incorporating  formal  models 
into  their  project  evaluation  decision.  As  models  differ  in  structure,  they  will 
differ  in  their  sensitivity  to  the  errors  in  different  estimates.   Figures  #1   and 
#2  suggest  that  the  same  estimates,  project  cost  in  this  example,  in  similar  lab- 
oratories can  still  have  widely  differing  error  characteristics.   It  is  therefore 
impossible  for  a  manager  to  assess  the  relative  value  of  different  selection  models 
in  his  own  laboratory  until  he  better  understands  the  nature  of  the  errors  inherent 
in  the  estimates  available  to  him.   This  inevitably  will  involve  each  organization 
in  a  study  of  its  own  estimating  procedures  and  capabilities. 
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